6 



PATENT ABSTRACTS OF JAPAN 

(1 DPublication number : 08-149464 
(43)Date of publication of application : 07.06.1996 



(51)Int.CI. H04N 7/24 

H03M 7/36 
H04N 7/01 



(21 Application number : 06-291 146 (71 Applicant : GRAPHICS COMMUN LAB:KK 

(22)Date of filing : 25.1 1 .1 994 (72)Inventor : KOMATSU SHIGERU 

KOBAYASHI TAKAYUKI 
SHINDO TOMOYUKI 
OKADA YUTAKA 
NAGAI NORIHIKO 
SAITO RYUJI 
KAWAMURA YOSHIKA 



(54) IMAGE DECODER PROVIDED WITH FRAME RATE CONVERSION FUNCTION 

(57)Abstract: 

PURPOSE: To display the reproduced image by being conformed to a display monitor 
even if the frame rates of an input bit stream and the data to be displayed and 
outputted are different. 

CONSTITUTION: When the image data after a decoding to be displayed next is 
already stored in frame memories 72 and 73the image data is displayed. When this 
image data is not storedwhether the image data before the decoding to be displayed 
next is within a buffer memory 7 and the reference image data necessary for decoding 
this image data is stored in frame memories 72 and 73 or not are decided. When the 
reference image data is storedthe data is immediately decoded and the image data is 
displayed. When all these conditions are not satisfiedthe image data displayed just 
before is redisplayed. 



CLAIMS 



[Claim(s)] 

[Claim 1]A device which decodes coded image datacomprising: 



Mode holding mechanism which holds it when the number of pictures per second in an 
input coding stream is in a relation slightly less than the number of display output 
pictures. 

A frame memory condition determining means which judges whether image data after 
decoding which should be displayed on the next present on display is already stored in 
a frame memory. 

image data before decoding which should be displayed on said next — the inside of a 
buffer memory — and a buffer memory condition determining means which judges 
whether image comparison data required to decode said image data is already stored 
in a frame memory. 

A redisplay means which carries out redisplay of the image data displayed immediately 
before if data required to display the following picture in said frame memory and said 
buffer memory by a decision result of said frame memory condition determining 
means and a buffer memory condition determining means was not assembled. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application]This invention relates to the image decoding device with a 
frame rate conversion function which can decode an image by a different frame rate 
from the coded picture signal. 
[0002] 

[Description of the Prior Art]When transmitting or accumulating the image data by 
which digital representation was carried outcoding is performed in order to reduce 
data volume. There is the method of lessening relative redundancy as the method of 
coding using the time or spatial correlativity of picture information (image data). 
[0003]There are some which code the difference of two continuous screens (frame) 
as a method of using time correlativityor detect a motion of a picture and perform a 
motion compensation, moreover — as the method of using spatial correlativity — a 
picture — the block (for examplea lengthwise direction.) of a predetermined size It 
divides the horizontal method into 8 pixels at a timeit carries out orthogonal 
transformation of the data within a blockcarries out scan conversion of the 
conversion factor (for exampleit rearranges in order of a high frequency component 
from a low-frequency component)and has some which perform variable length coding. 
The image coding system (it abbreviates to MPEG 2 hereafter) with which MPEG 
(Moving Picture Experts Group) is advancing standardization is using the two above- 
mentioned methods together. This is indicated to ISO/IEC 1381-2 entitled provisional 
advice "Generic Coding of Moving Pictures and Associated Audio" of MPEG 2. 



[0004]D rawing 3 is an example of composition of the conventional image decoding 
device. 

The decode processing means A and the display processing means B are comprised. 
The input bit stream 100 which is a bit string of the coded data is inputted into the 
decode processing means Aand the reproduced image 200 is outputted after decoding. 
The display processing means B performs a conversion process so that it may be 
suitable for a display monitor in said reproduced image 200and the generating picture 
300 is outputted. In MPEG 2the contents of processing which should be performed by 
the decode processing means A are specifiedand it has not set about display 
processing. 

[0005]D rawing 4 is an example of composition of the decode processing means A of 
the picture of drawing 3 . Decoding processing is performed by the buffer control part 
10the variable length decoder 20the scan conversion machine 30the inverse 
quantization device 40reverse DCT section 50and the motion compensation image 
restoration part 60 in drawing 4 . 71 is a buffer memory and 7273and 74 are frame 
memories (memory of the three frames mentioned laterlPand the B frame). As for 
dataand 83 and 84reproduction picture element data and 86 are B frame image data 
prediction frame data or prediction field dataand 85 81 and 82andas for the input bit 
stream for which 100 expresses the coded pictureand 200a reproduced image is 
shown. 

[0006]Nextoperation is explained. The input bit stream 100 is accumulated in the 
buffer memory 71 by control of the buffer control part 10 as the data 81. Variable- 
length decoding of the data 82 read from the buffer memory 71 is carried out by the 
variable length decoder 20. 

[0007]Although variable length coding of all the data is not necessarily carried outa 
fixed length code shall also be decoded with this variable length decoder 20. 
Nextinverse quantization is carried out by the inverse quantization device 40 after 
rearranging an order of data with the scan conversion machine 30. Nexta reverse 
discrete cosine transform is carried out by reverse DCT section 50. In the motion 
compensation image restoration part 60reproduction in consideration of a motion of a 
picture is performed. In MPEG 2a middle frame (here the B frame) is predicted in time 
to be a front frame (here the I frame) from both next frames (here p frames). 
Thereforeit is necessary to read the prediction frame data 83 and 84 of the I frame 
and p frames decoded beforehand to reproduction of the B frame from the frame 
memories 72 and 73 (in MPEG 2next p frames are decoded in advance of the B frame 
in time). There is field prediction other than the above-mentioned frame prediction in 
a prediction procedureField prediction as well as frame prediction predicts the middle 
field (here B field) in time to be the front field (here I field) from both next fields (here 
P field). The B frame or B field is reproduced in the motion compensation image 
restoration part 60 according to the prediction error which is an output of prediction 
frame data or the prediction field data 83 and 84and reverse DCT section 50and it is 



written in the frame memory 74 as the reproduction picture element data 85. The 
frame of IPand B in the frame memories 72 and 73 and 74 is read from each memory 
in predetermined order (B frame image data 86 is read in drawing 4 )and the 
reproduced image 200 is outputted. 

[0008]In the display processing means B of drawing 3 processing of the noise rejection 
of the reproduced image 200frame rate conversionetc. is made. 
[0009]Hereafterconversion of a frame rate is described. For examplethe movie 
comprises 24 frames in 1 second.This is codedand in order to decode again and to 
obtain PAL and a reproduced image of 25 frames per second (50 fields / second) like 
SECAMthe two following methods can be considered. 

[0010]The 1st method is a method premised on obtaining the reproduced image 200 
of 25 frames per second at the time of coding. Although this is not necessarily 
commonthe stream syntax top of MPEG 2/video is possible. It applies 3 1-time field 
circumference terms in 1 / 2 seconds at the time of decodingand one frame is 
decodedone frame is displayed the 3 fieldsit usually passes by a reproduced image 
output only thenand the 2 fields of other periods express it as it. A decoding start is 
made into an unequal intervalor the change of 2 field displays / 3 field displays is put 
in as information on the repeat first field (repeat.firstjfield) in a bit stream at the time 
of coding. 

[001 1]A frame rate is not changed at the time of codingbut it codes by 24 frames per 
secondand the 2nd method is decoded as it isand obtains the reproduced image 200. 
The decoding start time of each frame is regular intervalsand the information on the 
repeat first field is disregarded. In this casefor acquiring the picture of 25 frames per 
secondframe rate conversion must be carried out by the above-mentioned display 
processing means B. Using a frame memory the 2nd [ or more ] pagethe general 
method reads the frame before [ one ] being written in the 2nd page by 25 frames per 
second (interlace) for a displaywhile writing in the 1st page by 24 frames per second 
(PUROGURESHIBU). In that caseit is [ as seeming / reading processing / not to pass 
writing processing ] necessary to read an identical field to 1 / 2 seconds twiceand to 
adjust it to them at 1 time of the ratelike the 1st method. 
[0012] 

[Problem(s) to be Solved by the Invention]Since it is at the coding time and a display 
monitors frame rate is beforehand fixed to PAL/SECAM in the 1st method of the 
conventional technology mentioned aboveThe degree of display monitor option is 
lostand in the decoding device which connected the monitor of NTSC or a personal 
computereven if it can decode a bit streaman image cannot be seen. 
[0013]In the 2nd methodthe frame rate converter which carries the 2nd [ or more ] 
page of an expensive frame memory in the exterior of a decode processing means will 
be needed. 

[0014]An object of this invention is to cancel the fault of a device conventionally 
[ above ]and to be able to generate the reproduced image set by the display monitor 



which has connected by the decoding processing sideand for a cost hike to provide 

few image decoding devices. 

[0015] 

[Means for Solving the Problem]An image decoding device concerning this invention is 
provided with the following. 

Mode holding mechanism in which it judges it when an input coding stream has the 
number of input frames per second in a relation slightly less than an output frame 
numberand it holds the resultfor example by 24 frames per second like [ in case a 
display output is an object for 25 frames per second ]. 

Nexta frame memory condition determining means which judges whether image data 
after decoding which should be displayed is already stored in a frame memory. 
Nexta buffer memory condition determining means which judges whether image data 
before decoding which should be displayed is already stored in a buffer. 
A redisplay means which carries out redisplay of the image data displayed immediately 
before if data required to display the following picture in a frame memory and a buffer 
memory is not assembled. 

[0016] 

[Function]In this inventionwhen judging whether the image data of the frame which 
should be displayed on the next based on the decision result of a frame memory 
condition determining means has gathered when the value beforehand provided in 
mode holding mechanism is set up and not having gatheredthe output of a buffer 
memory condition determining means is investigated. If the image comparison data 
which needs the decoding previous image data of the frame which should be displayed 
on the next to have gathered in the buffer memory and decode it also exists in a 
frame memorywill perform decoding and display processing promptlyand if that is not 
rightBy the above-mentioned redisplay meansredisplay of the frame in front of a 
frame present on display is carried out. Simple frame rate conversion is attained 
without this newly forming a frame rate converter outside. 
[0017] 

[Example]The example of this invention is described using drawing 1 and drawing 2 . 
Drawing 1 is a block diagram showing the composition of one example of this 
invention. Drawing 2 shows the outline timing of decoding and the contents of 
processing which carry out a repeat display for the input bit stream for about 1 
second by this example. 

[0018]In this examplethe input bit stream is coded by the plog RESHIBU sequence 24 
frame per secondthe display monitor assumes the interlace 25 frame per secondand 
GOP (Group of Pictures) structured is explaining as BBIBBP (in B picturethe picture 
of six sheets exists by the following I picture from two sheets and I picture between I 
or P picture). 

[0019]In drawing 1t he same name and the number were given to the component and 



identical configuration element which were explained by drawing 4 of the conventional 
example. The component newly added from drawing 4 The mode register 61 in the 
motion compensation image restoration part 60the display clock generating part 62the 
vector decoding part 63the frame memory reading-and-writing control section 64 and 
the coded representation mode signal 11the buffer memory condition signal 12the 
24:25 conversion mode flag 65They are four signal wires of the display frame periodic 
signal 66. 

[0020]On the other handin drawing 2t he input bit stream 100 is input data to the 
buffer control part 10 shown in drawing 1 . The above-mentioned bit stream data 
stored in the buffer memory 71 with decoding processinglt doubles with a display 
monitor's frame period (1 / 25 seconds) from the buffer memory 71 It reads per frame 
and processing time until it writes regenerative data in the frame memories 7273and 
74 through the variable length decoder 20the scan conversion machine 30the inverse 
quantization device 40reverse DOT section 50and the motion compensation image 
restoration part 60 is shown. 

[0021]The writing of the frame memory in drawing 2 and read-out express the writing 
of a reproduced imagereference read-outand a display read-out sequence using an 
arrow for contents of processing of frame memory reading-and-writing control 
section 64 in applicable frame periodi.e.IPand B each picture of every. The arrow out 
of which the arrow with which the arrow of the direction included in each memory 
comes out of the writing of a reproduced image to a lengthwise direction comes to 
read-out as an image comparison and a transverse direction means read-out for a 
display. The reproduced image 200 is an outputted image obtained as a result of the 
processing mentioned above. 

[0022]Nextoperation of this example is explained using drawing 1 and drawing 2 . In 
drawing 1 through the buffer control part 10the input bit stream 100 is a rate of 24 
framesand is continuously stored in the buffer memory 71 in 1 second. Howeversince 
coding modes usually differ for every picture of IPand Bgenerated code amounts 
differ, the frame rate by which the buffer control part 10 is contained in a streamthe 
frame structureand a plog RESHIBU sequence ****** — etc. — it analyzes and tells 
the mode register 61 via the coded representation mode signal 11. The buffer control 
part 10 is provided also with the function as a memory information judging meansThe 
picture start code which shows the head of each frame of coding data to frequent is 
countedlt supervises whether the coding data for what frame is stored in the present 
buffer memory 71 and the frame memory reading-and-writing control section 64 is told 
about the result via the buffer memory condition signal 12. On the other hand in the 
motion compensation image restoration part 60an input from the scan frequency of 
the information from the coded representation mode signal Hand the display monitor 
under present connection by PUROGURESHIBU display 24 frame per second. And if a 
display output is 25 frames per secondsuch as PAL24:25 conversion modes will be set 
as the mode register 61 which functions as mode holding mechanismlt tells the buffer 



control part 10 and the frame memory reading-and-writing control section (it 
functions as a frame memory condition determining means and a redisplay means) 64 
by making the result into the 24:25 conversion mode flag 65. In this modeVBV delay 
(Video Buffering Verifier Delay) or DTS (Decoding.) as which the buffer control part 
10 specifies the read timing from the buffer memory 71 described in the stream It is 
not necessarily caught by the value of Time Stampfor exampleis referred to only at 
the time of a program receiving startand after itaccording to the display frame 
periodic signal 66it reads in order to decode coding data by an one-frame minute unit 
from the above-mentioned buffer memory 71. Usuallyvariable-length decodinginverse 
quantizationreverse DCTmotion image compensationetc. are processed in real 
timeand the read coding data is written in the frame memory respectively decided by 
the kind (IPB) of picture. Howeverto an input being 24 frames per secondsince an 
output is 25 frames per secondthe pause of decoding produces it at a rate of one 
frame at 1 second. This is the period when decoding processing was shown by x in 
drawing 2 . At this timethe frame memory reading-and-writing control section 64 
controls a reproduced image output according to the following priorities. 
[0023](D It will be displayed if the data for one frame of the turn which should be 
displayed on the frame memories 72-74 is assembled first (2) Nextthe coding frame 
data (usually B picture) of the turn which should be displayed on the buffer memory 
71 is together with all of one frameand displaydecodingif the image comparison 
required to decode it is equal to the frame memories 72 and 73. (3) If the above (1) 
and (2) is dissatisfiedit carries out redisplay of the frame displayed immediately before. 
[0024]As shown in drawing 2 when the coding data for one frame has not been stored 
yet in the buffer memory 71 While stopping decoding processing temporarilythe frame 
memory reading-and-writing control section 64Nextit is checked first whether the 
reproduced image data which should be displayed is already stored in the object for 
lor the frame memories 72 and 73 for P (since only I or P picture is decoded a priori 
and is already stored in the frame memory as an image comparison). If it existsthe 
frame will be outputted as the reproduced image 200. At this timeif it does not exist 
in the frame memories 72 and 73the frame memory reading-and-writing control 
section 64 outputs again the frame outputted immediately before for a display. The 
buffer control part 10 checks that the coding data for one frame has been equal to 
the above-mentioned buffer memory 71 and decodes the frame at the frame period 
after stopping decoding once. If this decode data is B picturethe frame memory 
reading-and-writing control section 64 will output it as the reproduced image 200 
promptly. On the other handif it is not B picturethe re output of the data of the frame 
displayed immediately before will be carried out. 

[0025]Although this example described the case where the display output of the input 
stream of 24 frames per second was carried out by 25 frames per secondsuch as 
PAL/SECAMIf the frame rate of input data is lower than an output frame rate about 
several or lesspercenteven if it applies to the apparatus which has a displaying frame 



rate original with a personal computera communication terminaletc.for exampleit will 
be satisfactory practically. Although BBIBBP also explained GOP structure by this 
exampleit is just going to understand that the same effect is acquired in other 
combination easily [ a person skilled in the art ]. 
[0026] 

[Effect of the Invention]As explained abovethe image decoding device by this 
inventionWhen the number of pictures per second in an input coding stream is in a 
relation slightly less than the number of display output picturesThe frame memory 
condition determining means which judges whether the image data after decoding 
which should be displayed on the next present on display as the mode holding 
mechanism holding it is already stored in the frame memoryThe image data before 
decoding which should be displayed on said next in a buffer memory And the buffer 
memory condition determining means which judges whether image comparison data 
required to decode said image data is already stored in the frame memorySince it has 
a redisplay means which carries out redisplay of the image data displayed immediately 
before if data required to display the following picture in said frame memory and a 
buffer memory by the decision result of said frame memory condition determining 
means and a buffer memory condition determining means is not assembledfor 
exampleSince redisplay of the same frame is carried out at the time of underflow and 
the stream coded by the plog RESHIBU sequence 24 frame per second can be 
outputted as reproduced images of 25 frames per secondsuch as a direct PAL 
systemby frame memory control and buffer controlSince it is also unnecessary to add 
to specify a display monitor beforehand at the time of codingan outside expensive 
frame rate converteretc.the increase in cost and circuit structure can be pressed 
down to the minimum. 

[0027]It is possible to change also by the case of 23.976 frames per second the input 
is accepted to be not by 24 frames per second but by MPEG 2without adding a hand 
in any way. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is a block diagram showing the composition of one example of this 
invention. 

[Drawing 2] It is a figure showing the outline timing of decoding and the contents of 
processing which carry out a repeat display for the input bit stream for about 1 
second in one example of this invention. 

[Drawing 3] It is a block diagram showing the example of composition of the 
conventional image decoding device. 

[Drawing 4] It is a block diagram showing the details of the composition of the decode 



processing means in drawing 3 . 
[Description of Notations] 

10 Buffer control part 

1 1 Coded representation mode signal 

12 Buffer memory condition signal 
20 Variable length decoder 

30 Scan conversion machine 
40 Inverse quantization device 
50 Reverse DCT section 

60 Motion compensation image restoration part 

61 Mode register 

62 Display clock generating part 

63 Vector decoding part 

64 Frame memory reading-and-writing control section 

65 24:25 conversion mode flag 

66 Display frame periodic signal 

71 Buffer memory 

72 Frame memory 

73 Frame memory 

74 Frame memory 

81 Data 

82 Data 

83 Prediction field data 

84 Prediction field data 

85 Reproduced image data 

86 B frame image data 
100 Input bit stream 
200 Reproduced image 



